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Abstract 


The purpose of this study is to examine individuals' computer based complex task performing 
processes and strategies in order to determine the reasons of failure by cognitive task analysis 
method and cued retrospective think aloud with eye movement data. Study group was five senior 
students from Computer Education and Instructional Technologies Department at a state 
university. A computer based complex task including a logical reasoning process was developed 
and utilized. Data was collected with cued retrospective think aloud containing a gaze video replay 
and eye tracking. All the retrospective protocols were transcribed for analysis then a coding schema 
was developed iteratively from segments. By drawing area of interest fields(AOI) of task panel, eye 
movement data in specific processes and fields were analyzed by using fixation duration and 
fixation count metrics. 

None of the participants completed the task successfully within the given amount of time (max-10 
min.). The findings yielded seven cognitive strategies (" information gathering-reviewing ', 
" identification well-defined instructions " "cue-seeking" "using cues" "assumption" "trial and 
error" and "crosschecking) and their actions obtained from the task performing process. 
Furthermore, this process is modeled by defining sequences and the relations between the actions 
and cognitive strategies. It has been revealed that trial and error is the most employed strategy 
and participants used trial and error without reasoning. Moreover, it has been revealed out that 
participants mostly failed in choosing and using the right strategy. 

Keywords: cognitive task analysis, retrospective think aloud, eye movements, complex task 


When approaching a complex cognitive task during problem solving process, individuals are reported 
to determine and apply various cognitive strategies on the way of reaching a solution. Determining 
successful and unsuccessful strategies is crucial when making instructional decisions. Moreover, it is 
equally important to describe what makes task a complex task. In his well cited study, Campbell 
(1988), reviewed the related research on task complexity and concluded that task complexity changes 
across personal characteristics (experience, familiarity, interests, working memory, span of attention 
and substantial cognitive demand) and task characteristics (multiple ways to arrive at desired end- 
state, multiple desired outcomes to be attained, conflicting interdependence among paths to multiple 
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outcomes and the presence of uncertainty). Web based environments or online learning environments 
can be considered as rather complex environments since they consist of various related or 
independent visual, audio or textual components. When individuals participate to these environments 
for various purposes and tasks i.e. learning, information seeking, reading or writing, there have been 
different intrinsic and extrinsic factor that effect to this process. 

In research related to performing complex tasks in learning environments, researchers generally focus 
on describing a successful process of task, modeling the process of successful individuals (i.e. experts) 
(Ertmer et.al., 2008a) and determining the differences between experts and novices (Ertmer et.al., 
2008b; Brand-Gruwel, Wopereis 8i Vermetten, 2005). However it is critical to determine factors that 
cause failure by examining thoroughly the reasons of why individuals become unsuccessful, at which 
stage and how frequent failure emerges and which processes cause them to fail. These findings would 
help to modify learning environments with regard to elimination of problems and training of 
individuals with strategies related to solutions of these problems. Thus, in this study, it was aimed to 
examine individuals' computer based complex task performing process and strategies in order to 
determine the reasons of failure by the cognitive task analysis method and cued retrospective think 
aloud containing eye movement data. In this section, first, what makes a task complex in learning 
environments will be outlined. Secondly, cued retrospective think aloud methodology through eye- 
movement replay will be explained. 

Complex Tasks in Learning Environments 

Complex tasks are cognitively demanding tasks in which individuals have to organize a large number 
of different types of cognitive processing, often not possible to make an automated or standardized 
response to a situation, which requires some predicting and comparing to choose a sequence of 
activity since there are usually several simultaneous goals to be met (Bainbridge, 1997). Marmaras 
and Pavard (1999) stated that complex task could be complex both at the cognitive level and at the 
level of the environment and these tasks require different types of problem solving such as decision 
making, diagnosis and planning, and complex cognitive activities such as anticipation, monitoring and 
mental calculations. 

Complex task environments have some common characteristics which are generally specified as 
(Marmaras & Pavard, 1999; Funke, 1991); 

• They consisted of interrelated and interacting dynamic components and factors. Complexity of 
the task as defined by the number of variables can be depending on also the degree of the 
connectivity among the variables and the type of functional relationship. 

• Semantic content of the task is rich. This richness of semantic content often reduces the 
uncertainty. 

• Events can occur at undetermined times and the nature of the problem to be solved can 
change 

• There is uncertainty regarding the time at which one or more events occur, and the severity 
of changes they cause to the work system. Not every action shows immediate consequences, 
effects often can occur with time delay. 

• There are multiple quantitative and qualitative objectives to be achieved, often conflicting, 
and with no predetermined hierarchy. Precision of goal definition, whether the goal is defined, 
whether there are multiple goals or are they contradictory. 

• There is a time limit in complex tasks which require immediate decisions. 

• Making a mistake could cause the successive mistakes. 
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Cognitive load has been argued as main characteristics of complex task performance (Sweller & 
Chandler, 1994) and it has been stated that the individuals can compensate for an increase in 
cognitive load (e.g., increasing task complexity) by investing more mental effort to maintain their 
performance at a constant level (Sweller, van Merrienboer, & Paas, 1998). 

Van Merrienboer, Kester and Paas (2006) suggested that in well-designed instructional environments 
extraneous load should be decreased and germane load should be optimized within the limits of total 
available capacity in order to prevent cognitive overload. 

Four key components of learning environments for complex tasks are claimed as (Merrienboer, 1997; 
van Merrienboer, Kirschner, & Kester, 2003; van Merrienboer, Kester, & Paas, 2006); 

• Learning tasks should based on real-life tasks and fulfill the role of a backbone for the training 
program. 

• There is supportive information available to learners to help them to perform the problem- 
solving and reasoning aspects of learning tasks. 

• Procedural information is presented to learners to perform the routine aspects of learning 
tasks. 

• Part-task practice are included to provide learners with additional practice for routine aspects 
of the complex task that need to be developed to a very high level of automaticity. 

Using Cueing Retrospective Think Aloud Method in Cognitive Task Analysis through Eye- 
Movement Replay 

Retrospective think aloud is a method which is based on gathering information by think aloud not 
concurrently but after the performance is over. Concurrent think aloud is stated to have some 
limitation especially when the task involves high cognitive load (i.e. complex tasks) since verbalizing 
information while performing tasks might distract subjects' attention and concentration and increase 
cognitive load. (Russo, Johnson and Stephens, 1989; Branch, 2000). Retrospective think aloud 
method is suggested to overcome these negative effects of concurrent verbal protocols (Guan, Lee, 
Cuddihy, & Ramey, 2006). Even so, since retrospective think aloud is also have been criticized for 
forgetting information and fabrication of process (van Someren, Barnard, & Sandberg, 1994; van Gog, 
Paas, van Merrienboer, & Witte, 2005), presenting visual cues belonging the task process stated to 
stimulate verbal reports of individuals (van Gog, 2006; Guan, Lee, Cuddihy, & Ramey, 2006). The 
general technique to stimulate retrospective think aloud is to present users a playback video of task 
session which facilitate the retrieval information from memory and provide veridicality which is an 
important issue for research related to memory processes and problem solving strategies (Hyrskykari 
et.al. 2008). 

Since eye movement provides an objective measurement of cognitive processes (Rayner, 1998), they 
might additionally facilitate reporting thoughts and eliciting comments from users when they are used 
as cue (Russo, 1979). It is suggested that containing information about information processed and 
area focused during the task, eyes provide a window into mental life and eye movement have the 
characteristics to continuously track both in space and time the evolution of mental events (Nakatani 
& Pollatsek, 2004; De'Sperati, 2003). However, since solely eye movement data is limited to show the 
experimenter where the participant looked but not reveal out the answer of the why questions with 
user's intention, it is important to support eye movement data with verbal reports (Seagull and Xiago, 
2001; Eger, Ball, & Dodd, 2007; Hyona, 2010). 
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Both of the verbal protocols and the eye movements reveal valuable information about cognitive 
processes. Therefore using them together provides both subjective and objective data to the 
researcher and allows the researcher(s) triangulate data to enhance validity of the findings. Already 
there have been usability and cognitive task analysis studies that applied these two methods together 
(Guan, Lee, Cuddihy and Ramey, 2006; Tobii 2009; (Jarodzka, Scheiter, Gerjets, & Van Gog, 2010; De 
Koning et al., 2010). Since cognitive task analysis is a process to determine cognitive activities 
required accomplishing a given task and yield information about the knowledge, thought processes, 
and goal structures (Chipman, Schraagen, & Shalin, 2000), choosing appropriate knowledge elicitation 
techniques to obtain the underlying cognitive processes and structures is important (Seagull & Xiago, 
2001 ). 

Cooke (1999) suggested three main categories of cognitive task analysis techniques as; (a) 
observation and interviews, (b) process tracing, and (c) conceptual techniques. In process tracing 
techniques, to increase the information yield for practical applications, protocols may be supplemented 
with retrospective review of videotapes with probe questions and eye movement data to provide an 
effective memory cue facilitating recall (Chipmann, Schraagen, & Shalin, 2000; Seagull & Xiago, 
2001). Guan, Lee, Cuddihy and Ramey, (2006) compared subjects' verbalizations with their eye 
movements and found that stimulated with gaze data RTA is valid and reliable method that provides a 
valid account of what people attended to in completing tasks, has a low risk of introducing 
fabrications, and its validity is unaffected by task complexity. 

Consequently, supporting and validating users' subjective verbal data which has limitations and 
deficiencies, with objective eye movement data can be regarded essential to provide rich and valid 
data (Hyona, 2010). From this point of view, this study aimed to investigate cognitive processes of 
individuals during a complex task performing process, executing a cognitive task analysis with 
stimulated retrospective think aloud containing a gaze video replay and eye tracking techniques to 
overcome user's loss of concentration and distracting cognitive resources problems. 


Method 

This study employed the cognitive task analysis to examine individuals' complex task performing 
process in a computer based task. Data is collected with cued retrospective think aloud containing a 
gaze video replay and eye tracking techniques. 

Participants 

In this study, purposive sampling procedure was applied as a qualitative sampling method. In 
purposive sampling, subjects are selected because of some their characteristics to define a distinctive 
subgroup from a small sample. This selection method which focuses on a particular unit or subgroup 
that is considered homogeneous, doesn't aim to generalization but providing information rich cases for 
in depth study (Patton, 1990; Marshall, 1996). 

Since only a single difficulty level complex task was used in this study, study group was selected from 
same department and the same degree class to fix complexity and difficulty of the task with respect to 
age, class, department, prior courses variables etc.. The complex task was consisted of problem which 
is similar to "Academic Personnel and Graduate Education Entrance Examination" questions. Due to 
the purpose of this study was to examine reasons of failure, especially novice participants, in other 
words students who have not experienced this exam and also have not started to study for this exam 
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yet, are considered. From this point senior students who have not entered but also proximate to 
enter this exam completing their undergraduate education and will practice their theoretic information 
in real life problems were chosen for this study. 

Accordingly, eight seniors from Computer Education and Instructional Technologies Department were 
selected randomly and interviewed to inform about the study. They were also asked if they started to 
study for "Academic Personnel and Graduate Education Entrance Examination" by solving any pilot 
exam tests. Six of the seniors were accepted to be volunteers to participate in this study and declaring 
that they have not started to prepare to the exam yet. One of the participants was excluded from 
study because of the problematic data. Consequently, study group is consisted of five senior students 
from Computer Education and Instructional Technologies Department who were considered as 
novices. 

Instruments and Materials 

Complex Task: A computer based complex task which is based on logical reasoning was used. The 
task consisted of a set of cues, an empty 5X5 matrix field and information items. A screen view of task 
is presented in figure 1: 



Figure 1. Screenshot of the Computer Based Complex Task 
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Problem solution field is comprised of 5X5 matrix. Using the cues and deducing the logical conclusions 
individuals are required the match five different nationalities with five different color houses, five 
different animals, five different car models and five different beverages. None of the car models, 
animals, beverages or house colors must not to match more than once. Participants are required to 
place the information items (right side) to the matrix field (on left side) by utilizing from the given 15 
cues (upper part). While some of the cues were well defined, having explicit certain statements, some 
of them were ill structured not containing precise information to go on but only usable after utilizing 
from well defined cues to make inference by making their relations with other cues. Participants had 
the flexibility to re-change information items' places or undo the actions. Totally 10 minutes were 
given everybody to solve the complex task. A pilot study was conducted with 3 graduate students, as 
experts. Since two of experts could solved the problem in given time, time is set to with ten minutes. 

Eye tracking: Tobii T120 eye tracker was used to record video of complex task performing process 
and eye movements. Eye tracker was integrated within the panels of the monitor and the tracking 
system had a 120 Hz sampling rate and an accuracy of 0.5°. Participants were seated 60-65 cm 
distance from eye tracking monitor and calibration was executed for each participants. Tobii Studio's 
RTA feature was used to playback to participants their task process video and to record audio of 
retrospective think aloud verbal reports. 

Data Collection Process 

Data were collected in individually sessions independently from others for each participant. A total 
experiment session was lasted in an average 40 minutes for each participant. 

Task implementation process consisted of three stages; 1) prior practicing and training individuals 
about original task (approx., 12min); 2) performing task (lOmin.) and 3) retrospective think aloud 
(approx. 15-20min). 

Pre-training and practice session: Individuals solved a pre-experiment task which is a simpler 
reasoning problem similar to the original task, to practice and become familiar with the solving 
principles of the experimental task. This practice problem is consisted of matching items and 
information cues as in original task but on a 4X3 matrix field. This task was executed paper-pencil 
based. Correctness or falseness was not taken into account in the analysis or anywhere of the study. 

After the pre-experimental task a demo tutorial video was watched to the participants that contained 
instructions about to solution of experimental task. This training tutorial video was prepared by using 
Captivate program and consisted of audio visual instructions and small examples about problem. This 
video took approximately 2.5 - 3 minutes and if the participants needed they are allowed to watch 
more than one times. 

Experimental complex task. After the pre-training session participants were seated to computer 
which has an integrated eye tracker and the task was opened on the computer. After the participants 
were calibrated they were cautioned to sit inactively so far as possible and informed about the task 
will be finished end of the 10 minutes. 

After the 10 minutes, researcher notified the participants time is up and terminated the program. 
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Retrospective think aloud session : After the task performance participants had a five minutes 
break. Then participants were instructed about directives of think aloud procedure. Researcher told 
that a recording of eye movements during their own task performing process will be shown and they 
are required to tell through what were you doing and thinking at the time. Participants were informed 
that red dots represent where they were looking and these dots become larger when/where they look 
longer. 

The playback control and the play speed control were given to the participants, so they had the 
flexibility of re-watching, stopping or forwarding. If the participants fell silent a long time, researcher 
intervened to think aloud. 

After the retrospective think aloud session participant were informed the experimental session has 
finished and were thanked for their participation. 

Data Analysis 

Data analysis process was conducted according to stages of Cresswell's (2007) data analysis spiral. At 
first all the retrospective verbal reports were transcribed into separate text files for each participant. 
Then all the transcription files were imported into QSR-Xsight qualitative data analysis program. Text 
data was disaggregated into segments by analyzing content data for similar patterns and grouping 
conceptually similar contents. After that a code scheme was developed by an iterative process by 
comparing all segments with others and with video records until a sufficient coding scheme developed 
gali§ilmi§tir (Glaser & Strauss, 1995). 

Coding scheme was developed to reveal out individual's task performing strategies and actions that 
are employed with those strategies. Although individuals' verbal reports constituted the primary 
source to inference strategies and actions, but also researchers checked those verbal reports from 
video recording for verifying inferential and making up missing statements. At first, 20 strategies and 
31 actions that are put into with those strategies were eliminated. After that, by continuing recursive 
analysis, some of the similar or overlap categories were gathered under same category and also some 
categories were excluded since found to be not reflecting the original action in the video record. As a 
result, after the iterative coding process seven strategies and nine actions that are employed with one 
of those strategies were determined. A flowchart of the algorithm of complex task process was 
developed that shows sequence of the actions and relations of with each other and complex task 
performance process was modeled with frequency of the strategies and the actions. 

Triangulation, member checking and peer review were employed for credibility and trustworthiness of 
the study (Lincoln & Guba, 1985). Triangulation of the method by conducting research with both of 
the qualitative and quantitative data provides a broader, deeper perspective, minimize the bias and 
enhance the validity, reveal the various dimensions of given phenomena (Perone & Tucker, 2003). 
Especially using the objective and subjective data together and supporting these data with each other 
will enhance reliability and validity of the research (Cresswell, 2007). In this study, data was collected 
by both retrospective think aloud and eye tracking for cognitive task analysis. Subjective verbal 
reports from retrospective think aloud was supported by objective eye movement data from eye 
tracking. 

Member checking is also another most crucial technique for establishing credibility which is based on 
verifying or testing the data, categories, coding scheme, interpretations or conclusion with participants 
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(Lincoln & Guba, 1985). Within the scope of this study, the process model of the task performance 
that includes strategies and actions of participants were presented and described one of the 
participants and asked whether this process model reflects his own task performance process. He 
confirmed that this model reflects his general task solution process. 

For the inter rater reliability peer debriefing was used to legitimaze the interpretations and intercoder 
agreement coefficient was calculated. Miles and Hubberman's (1984) formula (agreements / 
(agreements +disagreements)) was used to calculate intercoder agreement coefficient and found to 
be %94,5. The mean interrater reliability was adequate with respect to the suggestion that 70 per 
cent intercoder reliability is considered satisfactory (Miles and Huberman, 1984). 


Findings 

The findings of this study will be presented in two sections. First, task completion performance will be 
analyzed and narrated. Secondly, cognitive strategies and actions during complex task performance 
will be presented. 

Task Completion Performance 

The task contained 25 information items to be placed in a correct order in a given period of time. 
None of the participants completed this task successfully in the given period of time. The first 
participant (PI) matched all of the 25 information items but only 2 of them were in their correct 
places while others were incorrect. However, when the participant checked his answer with cues he 
realized he made mistake. Due to the time limitation, he did not have time to go back to correct his 
choices. 

The second participant (P2) placed 14 of the information items on the matrix, among which 11 of 
them were on correct but 3 of them were on incorrect positions. The third participant (P3) placed 21 
of the information items on the matrix, among which 5 of them were on correct position, yet 16 of 
them were on incorrect positions. The participant four (P4) placed 15 of the information items on the 
matrix, among which 9 of them were placed on their correct position, but 3 of them were not. 
Similarly, participant five (P5) could not complete the task in the given time period (ten minutes). P5 
placed 12 of the 25 information items on the matrix, among which 6 of them were placed on their 
correct positions. 

Cognitive Strategies and Actions during Complex Task Performance 

The common cognitive strategies with the actions of these strategies were determined by cognitive 
task analysis during a complex task solution for five participants. 

Cognitive strategies are defined as procedures or mental routines that support individuals as they 
develop internal procedures which enable them to perform higher-level operations or accomplishing 
cognitive goals (Rosenshine, Meister and Chapman, 1996; Dole, Nokes and Drits, 2008). As well as 
determining cognitive strategies and action of participants, their eye movements also recorded to 
verify verbal reports and to facilitate participants' think aloud. To clarify where the participants more 
and longer fixate on and for which strategy they use which part, four main area of interests (AOI) 
were driven on the problem field: "cue field", "placement matrix"," information items" and "rules". The 
drawn AOIs of the problem field are presented in Figure 2: 
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Figure 2. Area of Interests of Complex Task Field 

Seven cognitive strategies (" information gathering-reviewincj ' , "identification well-defined 

instructions", "cue-seeking", "using cues", "assumption", "trial and error" and "crosschecking”) were 
obtained from the task performing process. In addition eight actions were observed related to 
strategies. 

The frequency of these strategies and their related actions are presented in Table 1: 


Table 1. Frequency of the Strategies and their Related Actions 


Strategy 

Action 

Frequency of 

Frequency of 



action 

strategy 

Information gathering- 
reviewing 

Identifying problem/task 

8 

8 


Seeking for well defined cue 

13 


Identification well-defined 

statements. 

13+7= 20 


instructions 

Choosing/placing well defined 

7 


cue statements. 



Cue-seeking 

Scanning 

Relating 

4 

5 

4+5= 9 


Relating 

10 


Using cues 

Placement 

16 

10+16= 26 

Assumption 

Placement 

9 

9 

Trial and error 

Placement 

21 

21 

Crosschecking 

Detecting mistakes 
Confirmation 

16 

5 

16+5= 21 
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Complex task performing process is modeled by defining sequences and relations between the actions 
and cognitive strategies based on verbal report and observations of video records (Figure 3). 



Figure 3. Complex Task Performing Process 
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As seen in figure 3, three main processes were emerged in complex task performing. The first process 
was described as "information gathering and problem comprehension" process with the use of 
information gathering-reviewing strategy. Second process was the "searching and selecting cue" 
process, which is divided into two sub-processes: 1) "starting a searching and selecting cue process 
with well defined instructions" with the use of identification well-defined instructions strategy; 2) 
"directly searching- relating selecting across all the cues" process with the use of cue-seeking, using 
cues, assumption and trial error strategies. Individuals started with one of the sub processes, but 
sometimes switched between them. The third process was labeled as "placing-crosschecking" process 
with crosschecking strategy. 

Process 1: Information Gathering and Problem Comprehension 
Information gathering-reviewing strategy 

By this strategy, participants read the given rules or other task related things i.e cues, information 
items etc. to understand what they were required to do and to comprehend problem logic not taking 
much action. This cognitive process occurred generally at the first 0.00-1.30 minutes after task 
started. This process is followed with identifying task/problem action. Example statements were 
presented below from this process that reflects how the participants verbalized he/she used this 
strategy and attempted to identify problem/task: 

PI: "Here I mused on solution of the task looking the cues as German's car, English..... 

At first, I did not already do too much thing because I tried to comprehend the logic of 
the problem. I focused on the rules" 

P2 : "In the first place I read the givens and look over task components. 

Then I tried to understand what there were in the cues. And I looked over other givens, 
information boxes at the right... " 

P4: "Here I am checking the cues to see what they are standing for. I tried to memorize. 

Then I looked to the drag and drop information boxes. I realized. ..." 

"Identifying the task/problem" action by the "information gathering-reviewing strategy" was 
articulated 8 times by three of the participants. PI used this strategy four times, P2 used two times 
and P4 also used two times along the task solution process. 

As seen from above example statements participants generally take a look at to the givens to them, 
read the cues to comprehend and investigate the other given information items i.e. information boxes, 
matrix field, rules... Only PI re-reviewed the rules more than one times to remind himself directives of 
task. 

Derived from the information gathering-reviewing strategy process, fixation durations on area of 
interests of also support verbal report of participants (Table 2). 
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Table 2. Fixation Durations and Total Duration Times in Seconds per Participant 
for Information Gathering-Reviewing Strategy 



Cue 

field 

Placement 

matrix 

Rules 

Information 

items 

Duration 

(sec) 

PI 

38,759 

1,749 

4,747 

7,110 

53 

P2 

6,639 

2,179 

0,383 

4,89 

18 

P4 

16,031 

7,029 

0,000 

6,6 

32 


Fixation Duration 



AOI 

Figure 4. Fixation Duration in Each AOI in Information Gathering-Reviewing Strategy 

As seen in Table 2 PI, used information gathering-reviewing strategy for four times, had the longest 
time (53 sec) for this strategy. P2, who used this strategy for two times, spend 18 sec. and P4 who 
also used this strategy for two times spent 32 sec. Total fixation duration chart (Figure 4) already 
shows that PI had the longest fixation duration following by P4 and P2 had the least fixation duration 
time for this strategy. 

Fixation duration per each AOI in figure 4 shows that the participant mostly fixated on the cue field. 
Also, individuals' verbal reports were in the support of their eye movements since they stated that 
firstly they looked at the cues and information items and then read the cues in depth to understand 
clues. Succeeding the cue field participants second most fixated on information items. However they 
also reported that since they are attempting to identify task problem, they needed to check embedded 
information in cues from information item boxes part or taking a glance at those information items. 
But, since they tried to comprehend cues by reading but only taking a glance to scan at information 
items, fixation duration on the information items is pretty fewer than the cue field. Placement matrix 
area and rules area were found to be least fixated on. This can be due to only PI read the rules but 
not two other participants and none of the participants not started to place any information. 

Apart from these three participants, other two participants did not report they started with this 
process; they started directly placing information on the matrix with second process. 
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Process 2: Searching and selecting cue 

After the information gathering and problem comprehension process participants started to second 
process which is the selection process to place information after reading cues. This process divided 
into two sub-processes: 1) "starting the searching and selecting cue process with well defined 
instructions" with the use of identification well-defined instructions strategy; 2) "directly searching- 
relating selecting across all the cues" process with the use of cue-seeking, using cues, assumption and 
trial error strategies. 

In the second process, two of the participants started the cue searching and selecting process with 
well defined instructions while other three started directly searching- relating selecting across all the 
cues. Eye movement related to second process also verified the two different starting strategies. 
Heatmap belongs to process of four participants' placing first three information item is presented 
below (Figure 5a, 5b, 6a, 6b) 






Figure 5a. Process Regard to Placing First Three Figure 5b: Process Regard to Placing First Three 
Information Items Starting with Well Defined Information Items Starting with Well Defined 
Items(P4) Items (PI) 



Figure 6a. Process Regard to Placing First Three 
Information Items Starting without Well Defined 
Items (P2) 



Figure 6b. Process Regard to Placing First Three 
Information Items Starting without Well Defined 
Items (P3) 
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Figure 5a and 5b show the process of two participants who started with well defined instructions and 
Figure 6a and 6b show the process of two participants who started directly with all the cues without 
discriminating the well defined or not. 

Three information items placed by first two participants (5a, 5b) are being well defined ones; however 
the first three information items placed by the other two participants (6a, 6b) are the not well defined 
and placed indefinitely. 

As seen in figure 5a and 5b, participants, who started with well defined instruction, focused and 
fixated mostly on cue field and read all the cue statements to select well defined items among others. 
They only scanned the information items area to drag selected information box not fixating longer 
here. One of the participants, P4, being started with well defined instructions reported his/her process 
as: 


"The thing I have attended now is the item which I found a cue, it is the part that I 
thought it stands me in good stead. That choice. I looked for, how I can replace the 
value of that choice . 

Here it was the first cue (9. item) which I determined as could be useful for me, it was 
also an item which includes direct and dear a cue statement. I think, at this moment I 
am looking for the Norwegian to place it on the matrix. 

There, I put it to its place with drag and drop and I marked that cue to indicate that it is 

done. 

At that moment, I am looking for an another cue which is direct and exact to place as 
prior , there. I found an item and this was the item which I decided clearly through the 
Norvegian, I also put it to its place. " 

On the other side, as seen in figure 6a, P2 read only first three cue statements to place first three 
information items without being sure accurateness of this placement. So, the initial cues were the 
most focused area for that person. Also P3, started without well defined instruction as seen in figure 
6b, read firstly the 9 th cue ( Norwegian lives in the first house) which was given in the example demo 
before the task and placed the information item given in that cue. After that he/she focused on 
information item boxes to find information related to that he placed (Norwegian-first house) and 
placed another two information randomly not being accurate (car, beverage....). As a result he/she 
focused mostly on the information items area but looked over the cues only for reading the 9 th cue. So 
the participants who did not started with well defined instructions focused different area not reading 
at first all the cues and comprehend them. One of the participants, P3, who did not start with well 
defined instruction, stated his/her process as: 

7 have started to fill the matrix. I put the Norvegian at the first field. I put it at the first 
field utilizing from the "Norvegian lives in the first house" cue. After that I browse the car 
brands again in order to find who uses what, which car brand. I entered the all nations 
first, one by one. I did not act upon any rule, I put all of them being law unto myself. 

Then, I placed the German into the second column. I placed the Danish into the third 
column, Sweeden into fourth column and English to the fifth column. After that I looked 
over "English lives in the red house" cue. I looked "Danish drinks tea" section. Then, 
utilizing from the "Sweeden feeds dog" cue I placed the dog item under the Sweeden. 
Utilizing from "Danish drinks tea" cue, I placed the tea item under the Danish... " 

Lastly, although P5 also did not start with well defined instructions and placed randomly, he/she was 
not taken into account in that analysis. Because of unclear process of him, not doing anything at the 
first 2.5 minute, reading only cues, and did not think aloud in that process. 
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Process 2a: Searching and selecting cue with well defined instructions process 
Identification. of_ well-defined, instructions strategy 

In this process, participants either identified firstly well defined instructions to place these clearly 
precise items or searched for other well defined precise item from cues, after placing well defined, to 
sustain this process. 

So, the actions performed with this strategy were determined as "seeking for well defined cue 
statements" and "choosing/placing well defined cue statements". Some of the examples verbal reports 
form participants were presented below 

P4: "Now, I am looking for a condition between items which could guide me to act in no 
uncertain terms not by chance. . 

Reading through and through, I am seeking which one is useful for me" 

P2: " Henceforward, I was looking for exact items to place on matrix and I was placing the 
indefinite ones at the right side. Because, I thought this is the only way to could get out. Even 
though I was late for this... " 

P5: "at the end, I placed the well defined items on the matrix. I read all the cues again from the 
beginning. " 

P3: "I put the Norvegian at the first field. I put it at the first field utilizing from the "Norvegian 
lives in the first house" cue" 

As seen above examples, identifying well defined instructions, participants either selected those and 
placed or re-searched for other well defined instructions. 

Identifying well defined instructions strategy was occurred totally 13 times for "seeking for well 
defined cue statements" and 7 times for "choosing/placing well defined cue statements". 

Process 2b: Directly searching - relating selecting across all the cues 

In this process participants used all the cues not discriminating well defined or not, conditional or not 
conditional. Also, after placing the well defined items and not finding anymore, participants tried to 
make the relations between cues to continue in this process. 

Cue seeking strategy: 

This strategy was employed by participants for seeking information among others that facilitates their 
process, guides to them or helps them to progress, so they can plan the next steps or obtain further 
data. 

By using this strategy participants either scanned the information items or related the cues with other 
data to determine next step. So the actions performed by this strategy are determined as scanning 
and relating. 

Two example statements are presented below that show participants scanned for information through 
the cue seeking strategy: 
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P2: "Here I reviewed all the cues again, and read again. Thinking that can it be 

something unnoticed or can I find a new cue?" 

P3: "After that I looked to Alfa Romeo car, thinking that is there an information about it, 

I tried to find it among given cues. Then I take brief look at all of them. Firstly, I looked 
the animat they feed. Following that, I scanned the car brands, I looked over them. 

Another two example statements are presented below that show participant make relations between 
cues through the cue seeking strategy: 

PI: "Just than here I tried to make a relation between cars and beverages. At first I wanted to 
continue on Audi car model. But after that, since I could not much cue about it I go toward to 
the BMW and Ferrari models here. " 

P2 : " AH I do at that moment was looking for a cue among the data to make a relation. After that 
I tried make a relation in my own way, thinking that if I put that there, could it fulfill other cue. " 

However cue seeking strategy was used while in either actions scanning the cues and relating the 
cues, these two actions are differed from each other. While scanning the cues, individuals did not 
fixated too long anywhere but rather take a cursory glance, on the other hand while searching the 
cues for relating they fixated longer on specific cues or information items to make a relation finding a 
new cue. 

Although all participants used this strategy, their eye tracking metrics also signaled different outcomes 
when scanning and relating the cues. 

Table 3 represent the fixation duration per AOI, total duration times, frequency of actions, mean of 
fixation duration and mean of total durations during the scanning and relating via cue seeking 
strategy. 


Table 3: Fixation Duration and Total Duration (Sec) in Scanning and Relation Actions 
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As seen in Table 3 scanning via cue seeking strategy was occurred 4 times while relating via cu 
seeking strategy occurred 5 times. Totally 31 second was spent for scanning and 11 second was spent 
for relating, while the mean time for one scanning action was 7.75 sec. and 22.2 sec. for relating 
action. Similarly, participants' total fixation duration while scanning via cue seeking strategy was 29,62 
sec. and 105.18 sec. for relating via cue seeking strategy. Finally, mean fixation duration time for one 
scanning action was 7.4 sec. while this mean was 21sec. for relating action. All of these findings 
indicate that participants spent approximately three times more fixation and spent three times more 
time while relating cues as compared to scanning them. So it can be suggested that, while making 
relations, individuals focus more heavily on the task process by making more mental effort than while 
they scan something taking glance superficially. 

Using Cues Strategy 

This strategy is used by participants to relate their data with given information in cues without 
discriminating well defined or not, try to obtain new well defined instructions by identifying the 
relationship with other cues or to place the cues directly on the matrix with regard to any of read 
information. So, since participants placed items on the matrix directly with regard to any information 
in the cues or related their cues with others, "relating" and "placing" were determined the two of 
action which were performed by this strategy. 

An example of participants' relating action with using cues strategy is given below: 

PI: "Later on, thinking that a top down approach would be logical, I focused on the cue 
of where does English live. I read all the cues one by one. After that, I came toward to 
Norvegian. Here I could not decide on which one should I put first. I attempted to take a 
look at cues. Eleventh.. That is, the person feed horse who lives next to the house of the 
Ferrari model car user, ...I mean I tried to match uppers and downs... mostly tried to 
reason on them at first. " 

P2: "At that moment, I was mostly attempt to relate green and coffee, since there was a 
cue as the owner of the green house drink coffee. I was trying to connect this relation. 
Actually there was a very simple logic of this task. I only need the match given cues 
according to items on the right side. That is, miiiii, for example considering the cue of 
German uses Alfa Romeo... if I juxtapose the information items of Alfa Romeo and 
German, afterwards I could find a field on matrix to place them. " 

An example of participants' placing action with using cues strategy is given below: 

P3: "Starting from the point of, the person, who lives next to the house of Ferrari user, 
feeds horse, I put the horse there,... since German was next to the Norwegian. " 

PI: "Afterwards I tried make relation between colors and beverages. It was the cue 
" there is white house next to the green house" which prompted me to placed the white 
house here. Since green house was drinking coffee I placed the coffee under the green 
house. " 

As seen above examples from verbal reports of participants, using the cues they either directly placed 
information items or attempted to make a relationship between information in the cues. While relating 
the cues by using the cues occurred 10 times, placing the information utilizing from the cues occurred 
16 times. 
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Assumption strategy 

This strategy was used by participants as a method to speculate their next action by utilizing from the 
given cues, analyzing these cues and defining the relations between those cues and also to inexact 
hypothesizing or finding solutions by reasoning. In this complex task context, participants employed 
this strategy to be able to place information items and to speculate the possible placement action 
which will promote the process. By using this strategy, participants did not directly placed the 
information items but only made assumptions about the possible placement action. Some of the 
verbal reports from participants that reflect they used assumption strategy were exemplified below: 

P2: "Here I thought the relation of BMW in quite a while. That is, I thought were water 
and cat under the same or different columns, if the person who lives next to the house of 
cat feeder, uses BMW car and if the person who is the neighbor of the water drinker uses 
the BMW? Later, I thought about the fields which I could place those. I mean. ...could I 
place water and cat under the same column or must they be placed under different 
columns. " 

P4: "After that, again I tried to make a relation with beverages. I thought that blue house 
owner could drink tea". 

Assumption strategy was used totally 9 times. Sometimes participants either made assumptions and 
then by acting in this assumption placed items by trial and error or after making assumption since 
they are not sure exactly they returned to seeking information to make exact relations. 

Trial and Error Strategy 

This strategy was defined as the strategy of making the placement action not being sure about 
accuracy but only by experience acting in their assumption. Participants used trial and error strategy 
generally for examining the factual truth or falsity of their assumptions. They placed information items 
on the matrix not being sure the exact accurateness of their action. So placing is determined as the 
action that was performed by trial and error strategy. 

PI: " In that section I checked all the cues again. Using horse, feeding cat,... that's.... 
models of cars got my attention. After that there was a cue statement as "Audi model car 
user also drinks mineral water as well". I thought about that the person who drinks 
mineral water could be the Norwegian. I totally left my choice to chance. And I dragged 
the Audi there and thinking that he can also drink mineral water, I dragged the mineral 
water under Audi". 

P3:"After that I put the Ferrari under the Norwegian. And I put it there since only BMW 

car remained for English, since I thought BMW is more logical, ... to tell the truth I 

did not placed it so meaningfully." 

P5: "And now I started to strive at matching. Here I placed all the nations by chance. 

After that, I returned to cues and rethought about them." 

Trial and error strategy was performed totally 21 times in the study by the five participants. 
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Process 3: Placing - Crosschecking 

In this process, participant placed the information items which they selected after the second process. 
Placing action was employed by either of three strategies namely identifying well defined instructions, 
using cues and trial error strategy. One example from verbal reports of participants for each strategy 
presented below: 

Placing by identifying well defined instructions: 

P4: "Yes, I found an item which I could benefit. This a well defined and exact cue. This an 
information which I could place it to its own place by checking. The person who lives in 
the center house". 

Placing by using cues; 

P3: "/ put the mineral water under the Sweeden based on "the user of Audi drink mineral 
water" cue. 

Placing by trial and error; 

PI: "Here I thougth that Sweeden could feed dog. Because, no more much animal name 
remained. And I trust to my chance. 

Trial and error strategy was the most used strategy for placing action being performed 21 times, while 
followed by using cue strategy with the 16 times and the identifying well defined instruction was the 
least used strategy for placing items with the frequency of 7. 

Crosschecking Strategy 

After the placing action, individuals mostly applied to examine the accuracy of their processing or right 
after realizing their mistakes they checked on consistency of their actions with the information at the 
cues to find error source. This process was performed by the crosschecking strategy. This strategy 
was defined as the process of checking accuracy of the action by examining the consistency of the 
given cues and their placements. Participants used crosschecking strategy to correct their error when 
realized that they made mistake or to control their process until that time. 

"Detecting mistakes" and "confirmation" were revealed out as the actions that were performed by 
crosschecking strategy. 

An example statement was presented below that show detecting mistakes by crosschecking: 

PI: "I realized that I have made a mistake between cars. I thought about to change the 
places of Alfa Romeo and Audi, but when I returned to cues I thought this could be a 
error. But afterwards, when I rethought, it seemed that this was the most logic 
replacement. I wanted the replace at first here, but when I returned to cues, the error 
between beverages..... The Audi car user also drinks mineral water .. but BMW is next to 

the Alfa Romeo, immmm there was a conflicting here. I thought this would not be 

correct even I replace the Audi and then I put back both of them to their own places. " 
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P5: 1 surmised about not to place nations by chance. Returning to cues, I tried to place 
them again. I endeavored the place correctly. For this reason I undid all the nations, 
except Norwegian ". 

An example statement was presented below that show confirmation of action by crosschecking: 

PI: "Not to make a mistake I checked that the Norvegian lives in the first house again 
and again. Because I was thinking this was the way for me to solution. " 

P3: "Again, I looked over the Norvegian lives in the first house to check if it was correct. " 

By crosschecking strategy participant detected mistakes 16 times and confirmed their actions 5 times. 
This result showed that rather than checking and confirming step by step their process, participants 
generally detect their mistakes after they realized. 

As seen above examples crosschecking strategy was employed by participants only for making sure 
themselves and verifying themselves by reviewing their progress. On the other side, while they are 
detecting mistakes they go through various cognitive processes as undo the actions or changing the 
location of placed items on the matrix, identifying the error source by comparing the new information 
with old ones or thinking on correcting solution of their errors. So participants spent more time for 
detecting mistakes as against to confirmation and their eye movement also supported that finding 
(Table 4). 


Table 4: Total fixation duration and total time spent in confirmation and detection mistakes 
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As seen in Table 4 confirmation with crosschecking was occurred 5 times while detecting mistakes 
occurred 16 times. Totally 40 sec was spent for confirmation and 415 sec. was spent for detecting 
mistakes, while the mean time for one confirmation action was 8 sec. and 25.9 sec. for one detecting 
mistakes action. Similarly, participants' total fixation duration was 28.9 sec. while confirmation via 
crosschecking strategy and 355.24 sec. while detecting mistakes via crosschecking strategy. Finally, 
mean fixation duration for one confirmation action was 5.78 sec. while this mean was 22.2 sec. for 
one detecting mistake action. All of these results indicate that the participants fixated approximately 
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four times more while they are detecting mistakes as compared to confirming and spent three times 
more time while detecting mistakes as compared to confirmation via crosschecking. 

Ultimately, totally seven cognitive strategies were derived from the complex task performance of five 
participants. Apart from the specific processes of these strategies, to examine where the participant 
mostly carry out the cognitive processes fixation duration per AOI, fixation count and heatmap visual 
of all the task completion process was obtained (table 5, figure 7). 

Table 5: Total fixation count and fixation duration per AOI in general task completion process 
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Fiaure 7. HeatmaD of Task ComDletion Process of Five ParticiDants 
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As seen in Table 5, participant mostly and longest fixated on the cue field area. Following the cue 
field, placement matrix field was the second most fixated area by participants. Information items area 
was at the third rank and rules area was the least fixated area. Heatmap of the general task 
completion process also supported those statistical data indicating that the cue field and matrix 
placement was the most heavily focused by the participants. 


Discussion 

This study investigated the individuals' cognitive processes during a complex task performance. 
However all of the novice participants failed to complete the task successfully. A cognitive task 
analysis with cued retrospective think aloud through eye movements was executed to determine the 
cognitive factors of the failure. As a consequence of analysis, the task process was modeled by 
cognitive strategies and the actions of these strategies employed by the participants during the task 
performance. Two of the participants did not use information gathering and reviewing strategy to 
identify problem/task and started directly solving task. On the other side two of three participants who 
attempted to identify the task problem before starting solution spent rather a short time in that 
process while one of them focused longer in this process. Brand-Gruwel, Wopereis & Vermetten 
(2005) also found that individuals generally start to problem solving process with defining the 
problem; however, reading the task instructions more often to activate their prior knowledge experts 
spend more time than the novices on defining the problem. Since, all of the participants failed to 
complete the task with success in this study, one of the reasons of this failure can be seen as that 
they did not spend enough time to identify problem and comprehend the task or directly skipping this 
process. 

Some of the participants started to task with well defined instructions while some of them started 
without discriminating well defined. Nonetheless, all of the participants failed to complete the task 
successfully regardless of the different starting strategies. This could be because all of the 
participants, regardless of starting with well defined instructions or not, used trial and error strategy 
to progress in the process after a while. Time limitation can be the reason, why participants employed 
to trial and error strategy in the course of time despite they started identifying well defined 
instructions. Because it was found that participants who started with well defined instructions, focused 
and fixated more and spent more time to select well defined items among others while the others who 
started directly with all the cues, fixated shorter and spent less time without discriminating the well 
defined. Rayner (1978) also stated that longer fixation durations indicate an intensive cognitive 
processing or having difficulties in information processing. 

Trial and error was found to be the most and the identification of well defined instruction is the least 
employed strategy for placing. Considering the reasons for failure, it has been revealed that trial and 
error is the most employed strategy when deciding to place information. Users' verbal reports showed 
that participants used trial and error without reasoning. Similarly, this finding is in line with other 
research in that novice people tend to work with trial and error strategy (Breslow, 2001) and 
unsuccessful individuals used random trial and error approaches more frequent than successful 
individuals (Hackling, 1984).. 

Participants employed quite frequently crosschecking strategy but more frequently for detecting 
mistakes rather than confirmation. Although participants spent more time for detecting mistakes as 


22 



Turkish Online Journal of Qualitative Inquiry, October 2012, 3(4) 


against to confirmation, their eye movement also showed that, they fixated approximately four times 
more while they are detecting mistakes as compared to confirming. So it can be suggested that 
individuals make much mental effort during the detecting mistakes (Rayner, 1978). Since individual 
progressed in the task process without verifying their action and spent much time to correct their 
mistakes which they detected at quite later stages, this can be reason of why they failed to complete 
the task within the prescribed time. So it can be suggested that, especially in complex learning tasks 
which include various component and steps that are in interaction with each other, regular feedback 
must be given to individuals in the process to inform them about their progress in the task. In this 
way, individuals don't spend too much time and effort to correct their mistakes in the end. 

Examining the general task processing areas showed that participants most heavily focused on the 
cue field and secondly on placement matrix field. However, when this finding is taken into account 
from the perspective of cognitive processes, these areas of interests are the two fields on which the 
most heavily cognitive processes are executed. All the participants read the cues continuously, make a 
relationship between them and compare them to determine the next action. Finding regarding to 
cognitive strategies of this study also support this result. Because the strategies of the cognitive 
model i.e. scanning by cue seeking, relating by cue seeking, placing by using cues, discriminating well 
defined instructions, crosschecking, making assumptions generally and mostly requires the using the 
cue field and placement matrix. On the other hand, since individuals use information items fields only 
to drag the information after the cognitive selection process and drop them to the matrix field, not too 
intensive mental processes executed on it. Lastly, since demo video and example questions with 
instructions were given to participants to guide them, they did not need to utilize from rules too much. 
As a result participants did not fixate and focused on information items field and rules field too much. 
Taking into account this finding from the viewpoint instructional design of learning environments, it 
can be suggested that determining the most important areas for cognitive processes and make clear 
instruction and precise information on this areas could help the user to spend less time and less 
mental effort to progress. 

This study showed that although novices employ various cognitive strategies they still remain 
incapable to perform a complex task successfully under the time pressure. A lack of metacognitive 
knowledge of novices about cognitive strategies of their own process could be one of the reasons of 
this finding. Flavell (1979) stated that "cognitive strategies are invoked to make cognitive progress, 
metacognitive strategies to monitor it". Individuals with high metacognitive knowledge are aware of 
their own cognitive process reflecting on their task process (Hill and Hannafin (1997) and able to 
decide where, when and what strategies are likely to be effective in achieving a goal (Flavell, 1979). 
Also in this study, participant's verbal reports showed that they were not aware their own strategy, i.e 
while one of the participants was using inductive method to match the information cues with the 
items, he stated in his verbal report that he was using a deductive method. 

Novice people are suggested to adopt a suboptimal strategy and maintain it during the task, not 
switching between other appropriate strategies (Hill & Hannafin, 1997; Lazonder, 2000). On the other 
side experts are specified as being faster and better at selecting appropriate strategies, also executing 
a strategy correctly and switching between strategies swiftly when needed (Allen & McGeorge, 2011; 
Strobel, & Pan, 2010; Lazonder, 2000; Strube, 1991). Similarly in this study, participants as being 
novices could not used strategies effectively and could not switch between them correctly or swiftly 
under limited time. So they mostly used trial and error strategy and even they realized they made 
mistakes they did not changed their strategy. 
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Examining the five novice participants' complex task performance process in depth, a common 
cognitive process model was suggested in this study. However, individual problem solution processes 
and individual verbal reports showed the various differences within the participants. In further studies, 
besides making general interferences from the results of small samples, each individual can be 
examined as a case to show individual differences. 

Participants mostly failed in choosing and using the right strategy. Considering the reasons for failure, 
it has been revealed that trial and error is the most employed strategy when deciding to place 
information. Users' verbal reports showed that participants used trial and error without reasoning. It 
can be asserted that, if participants are provided with strategy instruction and raised their awareness 
about their current strategy by using attention guidance methods, they can become more successful. 
From this point of view, in the second phase of this study, it is aimed to train participants based on a 
strategy instruction model and examine their strategy and success during performing complex tasks. 
By doing so, it is intend to point at the importance of strategy instruction for novices to become 
successful (experts). 
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